{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "(monitoring-models-grafana)=\n",
    "# View model monitoring results in Grafana\n",
    "\n",
    "You can deploy a Grafana service in your Iguazio instance and use Grafana Dashboards to view model monitoring details.\n",
    "There are five dashboards.\n",
    "\n",
    "**In this section**\n",
    "* [Overview dashboard](#model-monitoring-overview-dashboard)\n",
    "* [Details dashboard](#model-monitoring-details-dashboard)\n",
    "* [Performance dashboard](#model-monitoring-performance-dashboard)\n",
    "* [Applications dashboard](#model-monitoring-applications-dashboard)\n",
    "* [Configuring Grafana dashboards](#configuring-grafana-dashboards)\n",
    "\n",
    "```{note}\n",
    "You need to train and deploy a model to see results in the dashboards.\n",
    "The dashboards immediately display data if you already have a model that is trained and running with production data.\n",
    "```\n",
    "\n",
    "## Model monitoring Overview dashboard\n",
    "The Overview dashboard displays the models endpoints of the selected project. Only deployed models with Model Monitoring enabled are displayed.\n",
    "Endpoint IDs are URIs used to provide access to performance data and drift detection statistics of a deployed model.\n",
    "\n",
    "![overview](../_static/images/overview.png)\n",
    "\n",
    "The Overview pane provides details about the performance of all the deployed and monitored models within a project. You can change projects by choosing a new project from the\n",
    "**Project** dropdown. \n",
    "\n",
    "The top row of the Overview dashboard is a summary of:\n",
    "* The number of endpoints in the project\n",
    "* The average predictions per second (using a 5-minute rolling average)\n",
    "* The average latency (using a 1-hour rolling average)\n",
    "* The total error count in the project during the selected time period\n",
    "\n",
    "The central table presents details about the endpoints in the project:\n",
    "* **Name** &mdash; the model endpoint name.\n",
    "* **Endpoint ID** &mdash; the model endpoint ID. Press it to drill down to the [performance details](#model-monitoring-details-dashboard).\n",
    "* **Function** &mdash; the MLRun function to access the model.\n",
    "* **Model Class** &mdash; the implementation class that is used by the endpoint (e.g., `SKLearnModelServer`).\n",
    "* **Error Count** &mdash; the total number of errors: includes prediction process errors such as operational issues (for example, a function in a failed state), as well as data processing errors (for example, invalid timestamps, request ids, type mismatches etc.).\n",
    "* **Sampling Percentage** &mdash; the sampling rate percentage. By default (100), all events are sampled.\n",
    "* **Drift Status** &mdash; no drift (green), possible drift (yellow), drift detected (red).\n",
    "\n",
    "The table does not reflect the selected time range, it just displays the model endpoints status in the last 24 hours.\n",
    "\n",
    "The graphs at the bottom are:\n",
    "* Heat map for predictions\n",
    "* Heat map for average latency\n",
    "* Time series chart that displays errors by model endpoint over time\n",
    "\n",
    "See [How to Read a Heat Map](#how-to-read-a-heat-map) for more details.\n",
    "\n",
    "### How to read a heat map\n",
    "Heat maps are used to analyze trends and to instantly transform and enhance data through visualizations. This helps to quickly identify areas of interest,\n",
    "and empower users to explore the data in order to pinpoint where there may be potential issues. A heat map uses a matrix layout with colour and shading to show the relationship between\n",
    "two categories of values (x and y axes), so the darker the cell, the higher the value. The values presented along each axis correspond to a cell which is color-coded to represent the relationship between\n",
    "the two categories. The Predictions per second heatmap shows the relationship between time, and the predictions per second, and the Average Latency per hour shows the relationship between\n",
    "time and the latency.\n",
    "\n",
    "To properly read the heap maps, follow the hierarchy of shades from the darkest (the highest values) to the lightest shades (the lowest values).\n",
    "\n",
    "```{note}\n",
    "The exact quantitative values represented by the colors may be difficult to determine. Use the [Performance Dashboard](#model-monitoring-performance-dashboard) to see detailed results.\n",
    "```\n",
    "\n",
    "## Model monitoring Details dashboard\n",
    "The Details dashboard displays the detailed real-time performance data of the selected project and model endpoint. Use the dropdowns to change the project and/or model endpoint.  \n",
    "The detailed model performance data can be used to fine tune or diagnose potential performance issues that may affect business goals.\n",
    "\n",
    "![details](../_static/images/details.png)\n",
    "\n",
    "The top row presents the same data as in the Overview dashboard. The graphs are:\n",
    "* **Drift analysis** &mdash; a calculated result named General Drift from the histogram data drift application. The drift value is calculated as the average of Total Variance Distance and Hellinger Distance. The default thresholds are 0.5 for potential drift and 0.7 for detected drift. For more information about the default drift analysis, see {ref}`model-monitoring-overview`.\n",
    "* **Overall drift analysis by time** &mdash; the application metrics, including Total Variance Distance, KL Divergence, and Hellinger Distance.\n",
    "* **Incoming features** &mdash; the performance of the features in the selected model endpoint based on sampled data points from actual feature production data. The graph displays the values of the features in the model over time.\n",
    "\n",
    "## Model monitoring Performance dashboard\n",
    "The Performance dashboard displays performance details in graphical format.\n",
    "\n",
    "![performance](../_static/images/performance.png)\n",
    "\n",
    "This dashboard has five graphs:\n",
    "* **Predictions/s**  &mdash; the average number of predictions per second, over 5-minute intervals (rolling)\n",
    "* **Average Latency** &mdash; the average latency over time in both 5-minute and 1-hour intervals (rolling)\n",
    "* **Errors** &mdash; the number of errors over time\n",
    "* **Predictions Count** &mdash; the number of predictions over time in both 5-minute and 1-hour intervals (rolling)\n",
    "* **Custom Metrics** &mdash; custom metrics defined in the serving function\n",
    "\n",
    "## Model monitoring Applications dashboard\n",
    "The Applications dashboard displays details of the selected metric.\n",
    "\n",
    "![applications](../_static/images/mm-applications.png)\n",
    "\n",
    "* **Predictions** &mdash; the estimated number of predictions pushed to the selected model endpoint.\n",
    "* **Metrics** &mdash; the number of times the selected metric was calculated for the selected model endpoint.\n",
    "* **Average Value** &mdash; the average value of the selected metric for the selected model endpoint.\n",
    "* **Latest Result** &mdash; the most recent metric calculation within the selected time range. If the selected metric is an application result, the chart also displays its status and type.\n",
    "* **Metric Summary** &mdash; a table summary of the metric results including the start and end infer time, and the numerical value. If the selected metric is an application result, the chart also displays its status, type, and extra data.\n",
    "* **Metric Value by Time** &mdash; the metric value over time. The time value is based on the end of the infer time window.\n",
    "\n",
    "## Configuring Grafana datasources\n",
    "Verify that you have a Grafana service running in your Iguazio AI Platform.\n",
    "If you do not have a Grafana service running, please follow <a href=\"https://www.iguazio.com/docs/latest-release/services/fundamentals/#create-new-service\" target=\"_blank\">Creating a Service</a> to add it to your platform.\n",
    " When you create the service: In the **Custom Parameters** tab, **Platform data-access user** parameter, select a user with access to the `/user/pipelines` directory.\n",
    "\n",
    "In addition, you will have to add access keys to your model-monitoring data source:\n",
    "   1. Open your Grafana service.\n",
    "   2. Navigate to **Configuration | Data Sources**.\n",
    "   3. Press **model-monitoring**.\n",
    "   4. In Custom HTTP Headers, configure the cookie parameter. Set the value of `cookie` to:\n",
    "   `session=j:{\"sid\": \"<YOUR ACCESS KEY>\"}`\n",
    "   5. Press **Save & Test** for verification. You'll receive a confirmation with either a success or a failure message.\n",
    "\n",
    "<img src=\"../_static/images/model-monitoring-datasource.png\" alt=\"Grafana Model Monitoring Datasource\" width=\"400\"/><br>\n",
    "\n",
    "\n",
    "## Configuring Grafana dashboards\n",
    "From Iguazio 3.5.3, the overview, details, and performance dashboards can be found under **Dashboards | Manage | private**.\n",
    "You can also import the latest dashboards versions by downloading them using the following links:\n",
    "\n",
    "**Iguazio 3.5.3 and higher**\n",
    " * {download}`Model Monitoring - Overview <./dashboards/model-monitoring-overview.json>` \n",
    " * {download}`Model Monitoring - Details <./dashboards/model-monitoring-details.json>`\n",
    " * {download}`Model Monitoring - Performance <./dashboards/model-monitoring-performance.json>`\n",
    "\n",
    "**Iguazio up to and including 3.5.2**\n",
    " * {download}`Model Monitoring - Overview <./dashboards/iguazio-3.5.2-and-older/model-monitoring-overview.json>`\n",
    " * {download}`Model Monitoring - Details <./dashboards/iguazio-3.5.2-and-older/model-monitoring-details.json>`\n",
    " * {download}`Model Monitoring - Performance <./dashboards/iguazio-3.5.2-and-older/model-monitoring-performance.json>`\n",
    "\n",
    "Upload dashboards to your Grafana service by:\n",
    "   1. Navigate to your Grafana service in the Services list and press it.\n",
    "   2. Press the dashboards icon in left menu.\n",
    "   3. In the Dashboard Management screen, press **IMPORT**, and select one file to import. Repeat this step for each dashboard."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
